MODEL-NEMOTRON-H W2: the non-gated relu² MoE expert — one GEMM, not a merged pair (#517) - #528

…M, not a merged pair (#517)

Every grouped-MoE path in this tree is SwiGLU-shaped: a merged gate+up pair with
a silu(gate)*up epilogue. NemotronH's expert has no gate half at all --
`ckpt_names=("up_proj", "down_proj", "")` (nemotron_h.py:220 @ 555967922), the
empty third entry being the absent gate -- so the expert is

    h = up_proj(x); h = relu(h)^2; y = down_proj(h)

SEAM VERDICT: this is NOT a new merged pair and gets no `MergedGemmGroup`
descriptor. `MergedGemmGroup` describes N GEMMs SHARING operand A collapsed into
one launch; with N == 1 there is nothing to merge and no launch to save, so an
arity-1 descriptor would name a fusion that does not exist. `MlpGateUpMethodBase`
is likewise a merged [2I,H] gate_up seam with no pair to hold. The arm is
therefore the EXISTING grouped projection plus the activation we did not have --
exactly the shape the gated bf16 archs had before their pair was folded
(kMoeGroupedGemmBf16 + kMoeSiluMul). The reasoning is recorded next to the seam
it excludes, in merged_gemm.h. No parallel MoE path was added.

    up   : kMoeGroupedGemmBf16  | kMoeGroupedGemmNvfp4Marlin (W4A16 g16)
    act  : kMoeRelu2                 <- the only new kernel
    down : kMoeGroupedGemmBf16  | kMoeGroupedGemmNvfp4Marlin (W4A16 g16)
    comb : kMoeCombine(..., routed_scale)

vt::MoeRelu2 (OpId::kMoeRelu2, appended before kCount; CPU + CUDA) mirrors
ReLUSquaredActivation (layers/activation.py:609-628) as the fused-MoE path
reaches it: activation_without_mul("relu2") -> MoEActivation.RELU2_NO_MUL
(layers/fused_moe/activation.py:33,98) -> `F.relu(input, inplace=True);
torch.square(input, out=output)`. The DTYPE ORDER is the mirrored part, not an
implementation detail: upstream's kernel (csrc/libtorch_stable/
activation_kernels.cu:673-678) widens to f32, clamps at zero in f32, squares in
f32 and rounds ONCE on the store. No new f32 buffer is introduced -- the op
reads and writes the caller's dtype and only its arithmetic is f32.

routed_scaling_factor goes on the OUTPUT (apply_routed_scale_to_output=True,
nemotron_h.py:234), so vt::MoeCombine gained a trailing `routed_scale`
(default 1.0f -- every landed caller stays byte-identical, proven by a memcmp
test) that multiplies the routed sum BEFORE the shared term is added. That is
literally moe_runner.py:389-406 (`fused_output *= routed_scaling_factor`,
`shared_output` untouched) then :722-725 (`shared_output + fused_output`).
Upstream forces the ROUTER's factor to 1.0 in exactly this case
(layer.py:291-300), which is the opposite polarity from Laguna, which folds the
same factor into the router weights by linearity (laguna_ops.h:48).

group_size=16 NVFP4 -- the spec's named risk -- is SUPPORTED, not emulated:
MoeMarlinArgs already defaults to group_size=16 / mxfp4=false, and
cuda_moe_marlin.cu:7,115-129 consumes exactly that (group_blocks=1,
s_type=kFE4M3fn, num_groups=size_k/group_size); 32 is reachable only via the
MXFP4 branch. A test pins the default so a later widening cannot silently
re-point these experts.

RED first: the new test failed to build on `vt::MoeRelu2 is not a member of vt`
and `too many arguments to vt::MoeCombine`. Green after: focused 10/10 cases,
71/71 assertions, Status SUCCESS; clean-tree Release -Werror rebuild 395/395
ctest; Debug arm green on the MoE + op-parity suites (Release is NDEBUG).

Mutations executed and caught (restored and re-proven green after each):
relu instead of relu^2 (5 cases red), silu instead of relu^2 (5 red), the square
narrowed through bf16 before the store (2 red -- the bf16-in/f32-out arm is what
sees it; a bf16-out-only test absorbs it), routed scale dropped (2 red), routed
scale applied to the combined output including the shared term (2 red), routed
scale folded into the router LOGITS (3 cases / 498 assertions red in
test_ops_moe_router_grouped), NVFP4 group_size default moved to 32 (1 red).

OWED, not claimed: this worktree has no GPU (nvcc absent), so the CUDA arms --
kMoeRelu2 on kCUDA and kMoeGroupedGemmNvfp4Marlin on the real g16 tensors -- are
compiled-and-reviewed only. The spec's W2 note records them as owed to a GB10
run. W3/W4 (loader, model file) still own wiring this into NemotronH itself.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]